What will I learn?

This tutorial is a primer, on how to go modular and use Docker 🐳 containers for your bioinformatics analysis tasks. More specifically, we will learn how to use the Deploit platform to assemble and deploy on cloud a reproducible and sharable bioinformatics workflow.

We will assemble the following resources into a workflow on the Deploit platform:

• 🐳 a Docker container, from DockerHub
• 🚀 a curated pipeline, available on the Deploit platform
• 📝 a 1000genomes project fastq.gz file, fetched from an EMBL-EBI FTP site

Last things first: What results will I get from the FastQC tool on Deploit?

After completing this mini workflow on the Deploit platform, you will have:
1) a plot-full FastQC html report,
with key metrics to assess the quality of your FASTQ file

2) sharable links to your Job Page s ,
with interactive plots and information about the resources and the results.

You can access the Jobs Pages from the sharable urls we created for this example:

• for retrieving the 1000genomes fastq.gz file : 🔗 wgetGunzipper
  
• for generating the FastQC html report: 🔗 fastQsee
  

and take a look at what a Job Page looks like below:

For every job run on Deploit, a Job Page report is created.

Why not install everything I need and run FastQC on my machine instead?

In principle, you could actually easily install the dependencies for running FastQC on your own machine. But this tutorial is more about learning how to use Deploit to easily combine resources (code from GitHub, containers from Docker Hub) to assemble multi-step bioinformatics workflows. This fastQsee pipeline will serve as our dummy example to go through the steps. Deploit enables you to structure your workflows as an assembly of individual, self contained units of computation (jobs), by bringing all the required resources to run an analysis (data, code, os & tools, computational resources) in one place. Each step in a bioinformatics workflow, will most likely utilize different tools and have different dependencies. But why not install all the tools on one machine, and run everything there, right? Well, for starters, dependencies!

We all know that it’s a hustle to make all the tools play nice together. There are several reasons why bioinformaticians have started joining developers and data scientists, and are slowly abandoning the monolithic, all-in-one place analysis environment approach and shifting towards more robust, modular and portable environment solutions. While virtualization and containers have been around for quite some time, Docker has really revolutionized the way we work the past few years. In a bioinformatics workflow, each process can utilize a different container as an execution microenvironment, with a main focus on preventing dependencies conflicts and ensuring reproducibility.

This modularity also unlocks many cool features as a positive side effect:

1) Portability : Installation (and hustle!) free run on any another machine
2) Cloudability : Easily deploy on cloud
3) Reproducibility : Allow for someone else to run the shame pipeline
4) Frictionless Retouching: Allows for easily removing, adding or retouching individual processes without affecting the rest
5) Isolation of Dependencies: Conflicting dependencies are isolated
6) Same tool, different tool version: Ability to use the same tool, but a different version of it in different processes if needed (legacy code in bioinformatics tools anyone?)

Why do I need a FastQC report?

The analysis journey of NGS generated FASTQ files should always start, as with any other data analysis task, with a robust E xploratory D ata A nalysis (EDA) bout.

The FastQC tool (Andrews S. et al, 2010), facilitates this task by providing a plot-ful html report, with key metrics for read quality.

FastQC quality assesment plots

list_of_img = c("https://github.com/cgpu/fastQsee_helper_repo/blob/master/images/per_base_sequence_quality.png?raw=true",
                "https://github.com/cgpu/fastQsee_helper_repo/blob/master/images/per_sequence_content.png?raw=true",
                "https://github.com/cgpu/fastQsee_helper_repo/blob/master/images/per_sequence_quality_scores.png?raw=true",
                "https://github.com/cgpu/fastQsee_helper_repo/blob/master/images/per_tile_sequence_quality.png?raw=true"
                )
slickR::slickR(obj     = list_of_img,
               padding = "10" ,
               width   = '70%',
               height  = 300
)

If not familiar with how to interpret the plots, you can start by taking a look at the good and bad quality sequencing data examples that are available in the FastQC tool official webpage. There is also a really great presentation on “RNA-seq quality control and pre-processing” by Mikael Huss you can check out here:

What resources do I need to generate a FastQC report on the Deploit platform? (spoiler hint: links!)

In this tutorial, we will bring all the required resources to generate a FastQC report on the Deploit platform. For the most part the resources we will need can be provided in the form of links, that point to the following four fundamental ingredients of any data analysis pipeline:

wget -O - https://lifebit.page.link/ftp_SRR062634_fastq_gz | gunzip -c > SRR062634.fastq 

STEP 0: Create a Project for your analysis tasks

For generating the FastQC report, we will deploy two jobs:
1) One for retrieving the file
2) One for running the FastQC tool.

It is advised to create a Project to host the individual tasks/jobs of a workflow. Think of the Project entity in the Deploit platform, as your parent directory for the project. There, you will have access not only to the data and code, but also to all the Jobs that have been run. You will have the ability to revisit, clone and deploy again the same jobs very easily.

Log in to your Deploit account and access the Projects section from the light bulb icon 💡 on the left of your screen. Click on the green New button on the right, and provide a Name and Description for your new Project.

We set up the Project for this example by filling in:

• Name: “fastQsee”
• Description: “Quickly generate a FastQC report”

You can find an overview of how these steps look on Deploit below:

STEP 1: Port the frolvlad/alpine-bash Docker container to Deploit

After you log in to Deploit, find the Pipelines section from the navigation bar on the left of your screen. We will create a new pipeline in the Pipelines > My Pipelines & Tools section.

This is how we will port the frolvlad/alpine-bash Docker container to Deploit, so that we can it use it for retrieving the fastq.gz file. Have the link to the Docker repository ready for copy+pasting
URL to Docker Hub: hub.docker.com/r/frolvlad/alpine-bash Have the link to the Docker repository ready for copy+pasting and then go ahead and click the green New button on the right.

As shown above, you will be prompted to select “Where are you porting your pipeline from?”. Click on the Docker whale and then click on Select to proceed. Continue by filling in the required fields to port the container. You can have a look at how we set this up below as an example:

• Docker hub URL: https://hub.docker.com/r/frolvlad/alpine-bash
• Name : “wgetGunzipper”
• Default command: (leave this blank)
• Description: "A lightweight Linux distro for wget/gunzip

The Default command is handy for saving time, but for now we will leave this blank so that we can use the command field as a terminal.

You can see an overview of the process described above in the following gif:

STEP 2: Use the newly created pipeline to retrieve and uncompress the fastq.gz file on Deploit

Time to use our newly created pipeline, and utilize the Docker container that we ported. We will use the command field on Deploit as a terminal, and the Docker container as our operating system to download and decompress the fastq.gz file we selected for this example. We will then deploy the job and come back to find our FASTQ file in the Data > Job Results section on the Deploit platform.

We will run the following command as mentioned earlier:

wget -O - https://lifebit.page.link/ftp_SRR062634_fastq_gz | gunzip -c > SRR062634.fastq

Have it ready for copy+pasting, and let’s go back to the Deploit platform. Access the newly created pipeline that utilizes the Alpine Linux docker container (we named ours wgetGunzipper) by clicking:

Pipelines > My Pipelines & Tools > wgetGunzipper

Paste the command from above in the Executable field. Take a look at how this will look on the Deploit platform below:

You can see an overview of the final command in the bottom of the screen:

Now we are ready to deploy the first of the two jobs for retrieving the FastQC report.

Click on Next on the top right of your screen. You will be redirected to the page where you will:

1. Select Project, in which your job belongs to (we selected fastQsee)
   
2. Select instance for deploying your job
   

When you select both, you are ready to submit your job. Go ahead and click Run job. You will be redirected to the Jobs page. Your job will be scheduled, initialized and be completed shortly. Take a look below to check how these steps should look in the Deploit platform.

After job completion, you can access the decompressed FASTQ file in the Data > Job Results section. We expect to find our FASTQ file in the fastQsee Project folder, with a filename as we defined it when retrieving it SRR062634.fastq.

Reminder: This is the command we submitted to download, decompress and rename the fastq.gz file:

STEP 3: Use the uncompressed FASTQ file as input to lifebitai/fastqc

Our input file for the next and last step, the uncompressed FASTQ file is now available on the Deploit platform in the Data > Job Results section. We will use this file as input for the lifebitai/fastqc pipeline, which is essentially a dockerized version of the FastQC tool and its dependencies. In the Pipelines > PUBLIC PIPELINES & TOOLS section, start typing "fastqc" in the search bar to easily find the lifebitai/fastqc pipeline.

Time to set up the pipeline. Let’s:

1. define input data
   
2. select Project that the pipeline will be associated with
   
3. select instance (cloud resources)
   
   

As shown above we have selected as input data, the 1000geomes FASTQ file we fetched from the EMBL-EBI FTP server in the previous step. None other parameteres are required to run the fastqc pipeline. We then selected an instance (anything larger than 1 CPU would work).

STEP 4: Job completed! Time to inspect the FastQC html report

Once the job has been completed, you can access your results from the Job Page, as shown below:

The Job Page serves like a summary report that includes information about:

• `` Pipeline and configuration```: Name of Pipeline and configuration selected for this job
• Job status: Job progress (% complete) or if it has failed
• Resource Monitor: CPU usage, RAM requirements
• Table overview: Runtime, number of processes, total cost

It also works as a portal to access all the generated output files in the Results section. Every job is assigned a unique Job ID, so that you can reference back to them or to retrieve programmatically information about the job using your private key through Deploit’s restful API.

STEP 5: Share, reproduce, re-use